feat(memory-pool): cross-machine zero-copy tensor transport for end-edge-cloud collaboration - #3079
feat(memory-pool): cross-machine zero-copy tensor transport for end-edge-cloud collaboration#3079tang-canran wants to merge 102 commits into
Conversation
…ensor transfer Extend classify_transport with a fifth parameter is_cross_machine. When true, the function returns NetworkZenohTransport regardless of GPU topology — data serialises and routes through the daemon's Zenoh channel for cross-host delivery. Add 4 cross-machine test YAMLs: cpu2cpu, cpu2cuda, cuda2cpu, cuda2cuda — each deploys sender on machine A and receiver on machine B via _unstable_deploy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- DaemonNodeEvent::WriteMemoryPool: node→daemon, carries tensor bytes + metadata for cross-machine forwarding - InterDaemonEvent::MemoryPoolWrite: daemon↔daemon via Zenoh - PROXY_POOL_DATA static: caches remote tensor data for local reads - Handle incoming MemoryPoolWrite by storing in proxy pool Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Complete data path for cross-machine tensor transfer: - DaemonRequest::WritePinnedMemory: node→daemon, carries tensor bytes - DaemonNodeEvent::WriteMemoryPool: daemon handler, stores in PROXY_POOL_DATA - InterDaemonEvent::MemoryPoolWrite: daemon↔daemon Zenoh forwarding - DaemonReply::PinnedMemoryData: daemon→node, returns proxy pool data - Control channel: hex-encode proxy data as Metadata with proxy_data key - read_memory_pool: detect proxy_data, decode hex bytes, return as tensor - write_memory_pool: serialize tensor after local write for cross-machine Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…forwarding Remove the complex Zenoh publisher management from the WriteMemoryPool handler. The PROXY_POOL_DATA storage and read-path fallback are complete and functional for same-machine cross-daemon testing. Zenoh cross-daemon forwarding of InterDaemonEvent::MemoryPoolWrite will be added in a follow-up PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
WriteMemoryPool handler now publishes InterDaemonEvent::MemoryPoolWrite
via a dataflow-global Zenoh topic (dora/{network}/{dataflow_id}/memory-pool).
All daemons subscribe to this topic at dataflow startup — incoming
events are deserialized and dispatched through the existing inter-daemon
event handler, which stores into PROXY_POOL_DATA for local reads.
- dataflow_memory_pool_topic(): new topic helper in dora-core
- spawn_dataflow(): subscribe to memory pool topic, spawn listener task
- WriteMemoryPool handler: publish via Zenoh for remote daemons
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… proxy path Remote receivers rebuilt the proxied tensor as a raw uint8 view because the WritePinnedMemory/MemoryPoolWrite chain only carried (bytes, size, device). Add dtype/shape to every hop so the receiver reconstructs the original tensor semantics: - InterDaemonEvent::MemoryPoolWrite / DaemonReply::WritePinnedMemory: add dtype + shape fields - WriteMemoryPool handler: store (bytes, size, device, dtype, shape) in PROXY_POOL_DATA via ProxyPoolEntry alias (fixes clippy type_complexity) - Rust node API: write_pinned_memory() takes dtype/shape, exposed as Parameter::String/ListInt when reading proxy_data Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… on degraded links - MemoryPoolWrite subscription listener moved out of the spawn handler: on a degraded inter-daemon link declare_subscriber() itself can block, wedging the daemon event loop (heartbeats + node replies included) — observed as the sender hanging on WritePinnedMemory forever - publish offloaded to a tokio::spawn with CongestionControl::Block and explicit error logs: a dropped publish silently strands remote readers with a never-ready proxy pool (observed on WAN link hiccup mid-transfer) - tcp listener: log frame size + first bytes when deserializing a DaemonRequest fails, so protocol drift is diagnosable Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ches The new MemoryPoolWrite variant left replay-node and the record/echo/ hz/info commands with non-exhaustive matches (E0004). Add explicit arms: replay-node and topic tools ignore the event (no-op/continue), matching their handling of OutputClosed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…cross-machine receivers register_memory_pool() now writes the registration tensor through WritePinnedMemory so remote daemons receive it via Zenoh (CPU receivers only; GPU pools travel via IPC handles, which the proxy path cannot carry). Pulls dtype/shape from tensor info and logs push failures loudly — a silent drop strands remote readers with a never-ready pool. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e-read - sender: re-push registration data every 500ms until consumed; pace writes ~20s to outlast receiver read latency under host contention - receiver: re-read (not zero-copy) each iteration — cross-machine proxy pools deliver fresh bytes per write Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e object header Cross-machine receiver previews showed the PyBytesObject header (refcount/type/len) instead of the tensor data: the dict's "ptr" used PyBytes::as_ptr(), which yields the object start. Switch to as_bytes().as_ptr() — the payload slice. Verified cross-machine (5090↔A100): 61.44MB transfers now reconstruct byte-identical tensors (sender preview == receiver preview). Also clamp the peer-claimed size to the actual payload length: the CPU tensor path builds (ctypes.c_byte * size).from_address(ptr), so an inflated claim reads past the heap block. The local DORADMA and GPU paths validate; the proxy path was the sole unguarded one. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two independent stalls kept the cross-machine example from completing more than the first frame (verified on 5090↔A100 over a WAN): 1. daemon: bincode::serialize of the 61.44MB MemoryPoolWrite payload ran inline in the daemon event loop — 3.2s per frame in debug builds (hundreds of ms in release) — blocking output delivery (next_require) and subsequent node requests until the event channels backed up and the sender's WritePinnedMemory hung forever. Move serialize + declare + put all into the spawned publish task. 2. sender.py: the trailing node.next() at the end of each iteration waited for the *next* iteration's next_require, which the receiver only sends after the *next* latency output — which this loop hasn't produced yet. Classic self-deadlock: sender stuck at the second next() while the receiver waits for the next latency. Drop it (keep the 20s pacing). 3. receiver.py: the memory-pool event trails the latency output on a WAN (separate topics, no ordering guarantee) and the registration re-push keeps old frames in the proxy pool — a read can return the previous frame (assert: expected 1, got 0). Retry the read until the expected frame arrives (each read consumes one proxy entry). Verified end-to-end: sender preview == receiver preview on all frames, 3-frame run completes with no errors. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… perf_counter time.perf_counter_ns() is CLOCK_MONOTONIC — its epoch is each machine's boot time, so t_received - t_send across machines is dominated by the boot-time difference (A100 up 34 days, 5090 up 5 hours → measured 0.00002 MB/s). Both hosts are NTP-synced (same timezone, identical wall-clock seconds), so time.time_ns() deltas are the true transfer time: 12.94 MB/s measured over the WAN. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dual-end real DORADMA pools replace the proxy-pool + hex roundtrip (11x gap: 12.94 vs 148 MB/s). register gains a `machine` param resolved via the coordinator (warn-and-skip if unresolvable); write forwards the full frame for the remote daemon to memcpy straight into the pre-registered pool under the seqlock protocol; read stays the unchanged zero-copy fast path; free tracks both ends. v1 scope: cpu2cpu_cross only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Strict reading of the requirement: when machine is specified but the coordinator cannot resolve it (or there is no coordinator), the whole register does nothing — no pool is created even locally — returns None for the caller to check, and never crashes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolution failure and remote-creation failure now behave identically (warn, no pool created, register returns None, no crash) — only the warning text differs so the two failure classes are diagnosable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ees it The synchronous register already guarantees the remote pool exists before any write. Lazy creation on write is a redundant side path and a leak source: a write-created pool is outside the free tracking (free events only reference registered pools), so it would never be released. Missing pool at write time is now a warn-and-drop-frame defensive case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
8 tasks: message types, coordinator ResolveMachine (store API already exists), daemon-A sync register with spawned ack wait (deadlock-free), daemon-B pool mirror + direct seqlock writes + dual-end free, python machine param, examples + local dual-daemon E2E + negatives, perf check. Includes the daemon->coordinator runtime request-reply mechanism (new pending-reply map + WS dispatch) needed by resolve_machine. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…achine Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e green) The message-layer additions broke exhaustive matches in the coordinator, node API, daemon and CLI/replay tools. Add stub arms (warn / not-yet- implemented replies / no-op) so every commit compiles; T2-T4 fill in the real implementations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the T1 stub with the real store lookup (get_daemon_by_machine); unknown machines resolve to found: false. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Distinguish a store failure from an unknown machine in the logs — matches the codebase convention of warning on persistence errors. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RegisterCrossMachinePool: resolve via coordinator, publish RegisterPool over the memory-pool topic, await the remote RegisterPoolAck with a 5s timeout in a spawned task (the ack arrives through the event loop, so awaiting on the loop would deadlock). Warn texts differ for resolution failure vs remote creation failure. Adds the daemon->coordinator runtime request-reply mechanism (COORDINATOR_PENDING + WS dispatch). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The reply JSON nests found under the externally-tagged enum variant
("ResolveMachineResult"), so the previous extraction always returned
false and the successful register path was unreachable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CoordinatorSender::send_event wraps its own envelope with a fresh id, so resolve_machine's pre-built envelope was double-wrapped and dropped by the coordinator parse. Add send_event_with_id (single envelope with the caller's request id) and send bare Timestamped bytes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-end free RegisterPool creates a DORADMA pool mirror (same layout as the node API) and acks; MemoryPoolWrite writes straight into the mirrored data region under the seqlock protocol when the pool is cross-machine (legacy proxy path unchanged otherwise); FreePool removes the mirror. Extracts publish_memory_pool_event for the ack/free publishing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The publisher's own subscriber received its RegisterPool echo, failed to mirror (EEXIST — the local node already created the pool) and published a false ok=false ack that deterministically beat the remote's ack, failing every sync register. Publish with Locality::Remote; gate RegisterPool on the machine_id match; guard the direct write against corrupt headers; move the 61.44MB memcpy off the event loop; init the mirror with an odd generation so readers wait for the first write. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…machine mirror machine=None (default) keeps the local path; machine="B" registers the pool cross-machine through the daemon (coordinator resolve + sync ack). Failure rolls back the local pool and returns None — never crashes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
register_cross_machine_pool transport errors went through ? — no local rollback (leaking the shmem + host pin) and a Python exception, breaking the warn-and-no-op contract. Merge both failure channels into the shared rollback helper and return None. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When a cross-machine pool is freed, remove it from CROSS_POOLS and publish FreePool so the peer releases the mirrored shmem (T4 handler). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Update JSON schema (9c29568) Co-authored-by: Dora Bot <dora-bot@phil-opp.com>
…aflow-scoped cross state, subscriber lifecycle, non-Linux clippy) - write_memory_pool now withholds its reply until the mirror daemon confirms the segment write (MemoryPoolWriteAck, seq-matched), so the send_output notification that follows the write can never overtake the tensor data and the receiver cannot return a stale frame; the example's 300s polling workaround becomes unnecessary. Publish failures and a 120s safety timeout fail the write loudly instead of hanging it. - Cross-machine registration now rejects pools larger than MAX_MESSAGE_BYTES (64 MiB, 1 KiB margin for framing) with a clear error in both the daemon and the python extension — previously such pools registered fine but every per-frame push silently failed, leaving the receiver waiting forever. - Cross-pool state (cross_pools, CROSS_REGISTER_PENDING) is keyed by (dataflow id, pool id) instead of pool id alone: every node process restarts its pool counter from zero, so a bare pool id repeats across concurrently running dataflows and could alias another flow's registration, ack routing, or free. - The per-dataflow zenoh subscriber task handle is retained and aborted on finish_dataflow AND on the failed-spawn path (it is spawned before the node build, so a failed spawn never reached finish_dataflow), stopping the task/session/event-sender leak and duplicate consumers. - cleanup_all keeps machine_id used on non-Linux (clippy -D warnings). Tests: dora-memory-pool 14/14 (incl. new cross_pool_state_is_dataflow_scoped), dora-daemon --lib 209/209; clippy -D warnings clean on daemon/memory-pool/ message/cli; fmt clean. The python-extension clippy lint errors are pre-existing (pyo3 deprecations/unsafe blocks, excluded from CI clippy). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The new variant was missing from replay-node's exhaustive match (CI Check + Clippy both failed on the same E0004). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ternative) The node's cross-machine write request now carries only (id, size) metadata; the daemon reads the tensor from the sender's segment (name resolved deterministically first — the register-time initial push arrives before the python's local registration lands — then the daemon table for explicit name= pools) and forwards it through the existing zenoh + commit-ack path. The node→daemon request is KB-scale, so the MAX_MESSAGE_BYTES (64 MiB) transport cap no longer applies: pools up to the 1 GiB registration cap transfer correctly, and the registration-time rejection added earlier is removed. Errors reply to the node instead of propagating: a handler error tore down the node connection and cascaded into a daemon disconnect (observed: 'pool X has no local segment to read the write from' killed the WS connection, and the reconnect's startup sweep then removed the just-created segment). Verified: same-host cross-daemon smoke (torch-gated) passes end-to-end with the new path; dora-memory-pool 14/14, dora-daemon --lib 209/209, clippy/fmt clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- The same-host cross-daemon smoke reads via the direct==true path and bypasses the MemoryPoolWrite/MemoryPoolWriteAck machinery entirely (only manual two-host runs exercised it), so the ack resolution is extracted into resolve_cross_write_ack() and pinned by a unit test: a stale seq resolves nothing, the seq-matched ack resolves exactly its own pending reply, and a failed mirror write surfaces as an error reply. - The read fast-path bail message now reports the actual wait window (0.5s for local pools) instead of the hardcoded 3600s. dora-daemon --lib 210/210, dora-memory-pool 14/14, clippy/fmt clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Covers the multi-daemon bring-up (coordinator + --machine-id daemons, --local-listen-port on one host, zenoh rendezvous), the YAML essentials (cross_machine env, _unstable_deploy machine/working_dir), the true-WAN ZENOH_CONFIG three points, the commit-ack and shmem-reference write semantics, measured numbers (LAN ~40, WAN ~4 MB/s; native ≤1 MiB on WAN, ROS 2 RTT-paced), and a cross-machine debugging checklist. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- The shared-memory-reference read (full-size allocation + copy) now runs inside the spawned publish task instead of synchronously on the daemon event loop — a 61.44MB frame previously blocked heartbeats, node replies, and output delivery for the duration of the read. Errors resolve the pending write reply (seq-matched) as before. - The mirror-creating daemon now enforces the same 1 GiB cap as the local side: create_cross_pool_shmem allocates size + data_offset in /dev/shm straight from the remote RegisterPool event, so a buggy or corrupted peer could previously drive an unbounded allocation (memory-exhaustion DoS). The error flows back through RegisterPoolAck. - The mirror header JSON is built with serde_json instead of format! interpolation: dtype/device arrive from the remote event (untrusted strings) and quotes/backslashes could corrupt or inject into the parsed structure. dora-daemon --lib 210/210, dora-memory-pool 14/14, clippy/fmt clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The README describes usage and behavior; measured throughput lives in design.md §5 (single source of truth, updated with the 2026-08-11 LAN/WAN runs). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cross-machine writes now bypass the zenoh relay when the mirror daemon advertises a data listener: - The mirror daemon runs a direct-TCP listener (port 7410, overridable via DORA_MEMORY_POOL_DATA_PORT) and reports it in RegisterPoolAck. - The origin learns the target daemon's address from the coordinator (ResolveMachine now returns the target's WS peer address, tracked at registration) and opens a persistent connection per endpoint. - Frames carry [magic][dataflow][pool][seq][size][data]; the mirror reads the payload straight into the mirror segment's data region under the per-pool async lock + seqlock (zero user-space copies on the receive side); the origin pays a single user-space copy (segment → send buffer). The commit ack still arrives via zenoh, so the pending machinery is unchanged. - Falls back to the zenoh relay when no endpoint is known or the direct send fails (dead connection dropped and lazily re-established). dora-daemon 210/210, dora-coordinator 122/122, dora-memory-pool 14/14, clippy -D warnings and fmt clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…P codec - The direct-TCP data listener's accept loop now sleeps 50ms on accept errors — a persistent error (EMFILE/ENFILE fd exhaustion) previously spun at 100% CPU. - serve_cross_data_frame is split into handle_cross_data_frame (frame parse + mirror write, no zenoh) and the zenoh ack publish, and the codec is pinned by a loopback round-trip test: send_cross_data_frame → handle_cross_data_frame over a TcpListener, asserting the payload lands in the mirror's data region under an even seqlock generation and the returned ack info matches (dataflow, pool, seq). The same-host smoke (direct == true) bypasses this data plane, so this is the new steady-state write path's first automated coverage. dora-daemon 211/211, clippy/fmt clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The data listener previously bound at daemon startup for every daemon with a machine id, even ones that never mirror a cross-machine pool. It now opens only when the first RegisterPool asks this daemon to mirror something (in the RegisterPool handler, after the machine gate), so non-participating daemons never open the port. The bound port is still advertised in RegisterPoolAck.data_port. dora-daemon 211/211, clippy/fmt clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The origin dials the mirror daemon's coordinator-visible WS source address, which is the wrong dial target under NAT, multi-homed, or same-host coordinator deployment (e.g. 127.0.0.1 when the daemon connects to a co-located coordinator) — the direct fast path would silently never engage there. The mirror daemon now advertises an explicit dialable address via DORA_MEMORY_POOL_DATA_ADDR (full ip:port, parsed with a warn on garbage), carried in RegisterPoolAck.data_addr; the origin prefers it over the derived address, falling back to the derived one otherwise. dora-daemon 211/211, clippy/fmt clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The write-commit protocol's last un-covered link was the ack publish itself: the resolver was pinned by a unit test, but the mirror-side publish -> zenoh transport -> origin-side subscribe -> deserialize chain was only exercised by manual two-host runs. The new test spins up two hermetic zenoh sessions (loopback TCP, no scouting) and drives the production publish helper (Locality::Remote, so a same-session subscriber would never see the put), receives the ack on the origin subscriber, deserializes with the production method, and asserts the seq-matched pending reply is resolved. Also removes a duplicated doc comment on serve_cross_data_frame. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The direct-TCP → zenoh fallback is the steady state on a broken link, so the per-frame warn flooded the daemon log for the whole outage. The fallback is now tracked per pool (CROSS_DIRECT_DEGRADED, keyed like CROSS_WRITE_PENDING): the first failed write warns with the error, subsequent failures stay silent, and the first successful write after a fallback logs recovery once. Pinned by a unit test on the note_* state machine. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… write errors
Two direct-TCP data-plane issues from review:
1. `data_offset + size` was an unchecked add on the wire-controlled
`size` (u64 read straight off the socket): a size near u64::MAX
wraps the sum past the bounds check, and a size-byte slice would be
constructed past the mapping (UB; a remote-triggerable abort in debug
builds). All three `data_offset + len` guards now use checked_add
(handle_cross_data_frame, write_cross_pool_data,
read_pool_segment_data), with the frame path rejecting with the ack
info.
2. A mid-frame payload read failure left the mirror seqlock odd and
published no ack, stranding the origin until the 120s commit-ack
timeout. handle_cross_data_frame now returns CrossFrameError
carrying the frame's (dataflow, pool, seq) once the header is
parsed, and serve_cross_data_frame publishes MemoryPoolWriteAck
{ ok: false } on that path so the origin fails fast. The odd
generation is kept deliberately (fail-safe: readers reject the torn
frame; the next full write self-heals) — rolling it back would mark
half-written bytes as a complete frame.
Tests: wire_size_overflow_is_rejected_not_aborted (u64::MAX size must
be rejected with ack info, generation untouched),
payload_read_failure_stays_odd_and_carries_ack (mid-frame drop leaves
the odd generation, carries ack info, next write self-heals).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The inline comment described a Drop-rollback of the seqlock generation that no longer exists: the generation deliberately stays odd (fail-safe, readers reject the torn frame, next write self-heals) — matching the NOTE on DirectMirrorWriter and the payload_read_failure_stays_odd test. The old wording would have misled a future maintainer into assuming rollback safety that is not there. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ross-write state The per-pool async write lock was keyed by the bare pool id. Pool ids repeat across dataflows (each node process restarts its counter), so two concurrent dataflows writing their own 'pool_sender_node_1' segments shared one lock and were needlessly serialized — the same key-scoping issue the human review flagged for cross_pools. The lock is now keyed by (dataflow, pool): writes to one segment stay serialized, writes to different segments of concurrent dataflows run in parallel. Regression test write_lock_is_dataflow_scoped pins the aliasing case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-keying the direct-write lock map by (dataflow, pool) fixed the aliasing but changed the growth bound: the old bare pool-id key was bounded (pool ids repeat across dataflows), the new per-dataflow-UUID key accumulates one entry per (dataflow, pool) for the daemon's lifetime — a slow unbounded leak on a long-lived daemon cycling many cross-machine dataflows. finish_dataflow now drains the finished dataflow's entries from the async write-lock map, the write-seq counters (same never-drained shape, predates this PR), and the degradation set. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The direct-TCP data plane's error path (serve publishes
MemoryPoolWriteAck { ok: false } when a frame's identity is parsed but
the write cannot proceed) had no automated coverage — the codec
round-trip covered the happy path, and the error path's zenoh ack
publish was only exercised by manual runs. The new test drives
serve_cross_data_frame with a size-overflow frame over a real zenoh
session pair (hermetic loopback, no scouting) and asserts the origin
subscriber receives ok=false with the matching (dataflow, pool, seq)
and the overflow error. This closes the last testable half of the
standing two-host coverage non-blocker; only the true two-host
transfer remains manual.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cargo-audit fails CI on the new advisory (2026-07-29): Unix BROWSER handling in webbrowser 1.2.1 allows browser argument injection. Bump to 1.2.4, which pulls in the fixed release plus its updated transitive dependencies (objc2-app-kit etc.). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The unpinned install-action 'latest' raced between runner caches: one of the two parallel CI runs per push picked up an older cargo-audit that crashes parsing the Cargo.lock's multi-target windows-sys (webbrowser's windows-only 0.48.0 target dependency — 'invalid Cargo.lock dependency tree: failed to find dependency: windows-sys 0.48.0'), while the other run's newer cargo-audit passed. Verified locally: cargo-audit 0.22.2 parses the lock cleanly and reports the webbrowser advisory resolved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
install-action's tool@version form did not pin the runner's cached cargo-audit, so the parallel CI runs raced: the runner with a stale cached release crashed parsing the Cargo.lock's multi-target windows-sys (webbrowser's windows-only 0.48.0 dependency), the runner without it passed. cargo install --version 0.22.2 removes the cache from the equation; 0.22.2 parses the lock cleanly (verified locally against the exact HEAD lockfile). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
9aa51a4 to
491dffc
Compare
- daemon/src/coordinator.rs: keep our ResolveMachine reply routing (ReplyRouteRaw + resolve_machine) — upstream does not have the cross-machine ResolveMachine command; their side of the conflict was comment wording only - apis/rust/node/src/node/mod.rs, runtime-api: take upstream's newer versions (our branch only carried synced upstream commits) - runtime/src/operator/mod.rs: take upstream's deletion (runtime refactor; memory-pool does not touch it) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Merging upstream/main overwrote apis/rust/node/src/node/mod.rs with the upstream version, dropping the write_pinned_memory and register_cross_machine_pool wrappers our python extension calls (the control_channel protocol methods survived the merge). Restored both wrappers after free_pinned_memory. Lower unwrap budget 162 -> 160 (upstream main removed unwraps). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
🤖 Automated review by Claude — this review is fully automated (no human wrote it) and may contain mistakes; please verify before acting. Follow-up on the two commits that landed since the last automated pass:
Previously-reported items all appear addressed in earlier rounds; the only standing non-blocker remains the true two-host cross-machine path, which can't be exercised in single-host CI by physical constraint. Generated by Claude Code |
|
Both points verified — the restore is confirmed, and the process note is addressed with a mechanical audit rather than trusting the conflict resolution: 1. Pub-symbol audit across the merge (
2. Green full build/test at head. CI run on The standing non-blocker (true two-host cross-machine path) remains as-is — verified on the real WAN cluster (direct-TCP 51.15 MB/s LAN / 9.99 MB/s WAN, full-byte parity 30/30), but stays outside single-host CI by physical constraint. |
Memory Pool: Zero-Copy Tensor Transport across Daemons and Machines for End-Edge-Cloud Robotics
(1) Design — new architecture for multi-daemon and cross-machine deployment
This PR extends the memory-pool transport to the deployment topologies real edge-cloud robotics actually run: multiple daemons on one host (sensor, perception, and inference pipelines as separate daemons on a robot or edge box) and daemon clusters across machines (edge ↔ cloud). The additions:
Same-host multi-daemon — direct read across daemons
pool_{node_id}_{machine}_{counter}) and reads it in place, bypassing the daemon relay and even the mirror push entirely (same-host direct detection). Native dora has no such path: outputs to a consumer on another daemon are pinned to the daemon relay (Readiness barrier (#2666) counts unreachable remote subscribers: fixed 5 s per-node startup stall + lost direct-zenoh fast path for local subscribers in multi-machine dataflows #2738), so the same topology costs 89 MB/s instead of ~5.8 GB/s (65×).Cross-machine — reliable large-frame data plane with GPU staging
tc qdisc add dev lo root netem delay 100ms(a 0.5 MiB frame hangs identically to 40 MiB). There is no working native path for WAN large frames.touched_by/ targeted cleanup) prevent cross-machine ID aliasing and clean up pools on every daemon involved — orphan sweeps are scoped to the daemon's own machine so a sibling daemon's live segments are never touched.Existing single-daemon behavior is unchanged; the event stream, zenoh, and daemon relay code are untouched.
(2) Performance: memory pool vs native dora vs ROS 2 (3 scenarios × 4 device pairs + cross-machine links)
Benchmark: 100 frames of 10000×512 int64 (40.96 MB), turn-based per-frame handshake,
Average transfer throughput(data_bytes / (t_received − t_send)), same payload and cadence on both sides. Native dora runs with the 256 MiB zenoh SHM pool configured (its best zero-copy configuration). GPU columns (cpu2cuda/cuda2cpu/cuda2cuda) are measured with the same benchmark; the WAN row is the cross-machine example's 61.44 MB frames (15000×512) on the same data path.Memory pool (MB/s, this PR):
Native dora (MB/s, same benchmark):
tc netem delay 100msreproduces the data-plane hang at any sizeSpeedup: 2.6× single-daemon, 65× same-host cross-daemon. On a 1-Gbps LAN the native relay already runs at ≈85% of line rate, so that link leaves little headroom (the pool is at 32% of line rate there — docker bridge/NAT and the per-frame handshake are the current limiters, not the link; we report it as-is rather than label it WAN). The claims that matter are the same-host 65× (reproducible on one machine without a network) and the WAN rows below.
Cross-machine landscape — ROS 2 network DDS on a real WAN (RTT 38.3 ms, n=12/size, byte-contract all green):
(3) Purpose and significance — enabling end-edge-cloud tensor pipelines
VLA models and world models are driving explosive demand for on-device inference [1–4]; end-edge co-computing is becoming the mainstream deployment paradigm, keeping data transport inside the LAN at 1–6 ms round trips versus tens-to-hundreds of milliseconds over the WAN to the cloud [10,11]. A robot pipeline in this paradigm moves tens-of-MB GPU tensors (camera frames, point clouds, VLA features) through several processes — sensor → preprocessing → inference — on the same host, and then to the edge cluster or the cloud.
This PR makes that pipeline zero-copy at every hop:
One API (
write_memory_pool/read_memory_pool) covers end, edge, and cloud with automatic device- and topology-aware path selection, turning message-passing deployments into shared-state pipelines without changing the dataflow description.